.\" SPDX-License-Identifier: CC-BY-SA-4.0 or-later
.\" SPDX-FileCopyrightText: 2025 grommunio GmbH
.TH exmdb_provider 4gx "" "Gromox" "Gromox admin reference"
.SH Name
exmdb_provider \(em Gromox Information Store
.SH Description
exmdb_provider is a component for http(8gx). It offers a plethora of
low-level functions (124 of them) for operating on mailbox stores. This
functionality is also exposed by way of a Gromox-specific network protocol on
port 5000. The aforementioned RPC functions transparently operate over the
network and may connect to a remote exmdb instance. In other words, this component contains
a process-local function API, a network client, and a network server.
.SH Configuration directives (gromox.cfg)
The following directives are recognized when they appear in
/etc/gromox/gromox.cfg.
.TP
\fBexmdb_force_write_txn\fP
(Developer option.) Perform all SQLite transactions as write transactions (i.e.
with exclusive locking).
.br
Default: \fIno\fP
.TP
\fBexmdb_deep_backtrace\fP
(Developer option.) Record and report transaction problems with a multi-level
backtrace instead of a single-level location indicator.
.br
Default: \fIno\fP
.TP
\fBexmdb_ics_log_file\fP
Log ICS/synchronization requests (and their results) to this file.
.br
Default: \fI(empty)\fP
.SH Configuration directives
The usual config file location is /etc/gromox/exmdb_provider.cfg.
.TP
\fBcache_interval\fP
The inactivity timer after which a mailbox's sqlite files are closed.
.br
Default: \fI15min\fP
.br
Lower limit: 1s
.TP
\fBdbg_synthesize_content\fP
When this directive is set to 1, missing content files will not be regarded as
an error and the respective attachment or property is delivered with a
replacement string. If set to 2, a replacement string is always delivered; mode
2 is useful for reducing the amount of data downloaded when debugging ICS.
.br
Default: \fI0\fP
.TP
\fBenable_dam\fP
When set to \fBon\fP, inbox rule processing is allowed to create Deferred
Action Messages (DAM). Furthermore, the "Deferred Actions" folder will have its
contents shown. / Conversely, if this directive is \fBoff\fP, no DAMs will be
created, and the DAM folder in inboxes is presented as empty to clients (even
if it has content from earlier).
.br
Outlook's DAM handling is poor and if you experience a crash with a primary
mailbox that is in non-cached/online mode a few seconds after Outlook has
opened it, turn this option off for mitigation.
.br
Default: \fIon\fP
.TP
\fBexmdb_body_autosynthesis\fP
When a client requests either PR_BODY, PR_HTML or PR_RTF_COMPRESSED, but that
property does not exist on a particular message, automatically synthesize the
data on-the-fly from another of the available formats.
.br
Default: \fIon\fP
.TP
\fBexmdb_file_compression\fP
Compress content files (bodytexts and attachments). Possible values: \fBno\fP,
\fByes\fP (zstd\-6), \fBzstd-\fP\fIlevel\fP (level=1..19).
.br
Default: \fIzstd\-6\fP
.TP
\fBexmdb_hosts_allow\fP
A space-separated list of individual host addresses that are allowed to
converse with the exmdb service. The addresses must conform to gromox(7) \sc
"Host addresses". No networks and no CIDR notations are permitted. This option
deprecates the /etc/gromox/exmdb_acl.txt file used before Gromox 2.8.
.br
Default: \fI::1\fP
.TP
\fBexmdb_listen_port\fP
The TCP port number for exposing the timer service on.
.br
Default: \fI5000\fP
.TP
\fBexmdb_pf_read_per_user\fP
Keep public folder read states per user (1) or keep one state for all
users (0).
.br
Default: \fI1\fP
.TP
\fBexmdb_pf_read_states\fP
When set to 0, messages in public stores/folders will always be shown as read
and the folder summary will reflect that.
.br
When set to 1, messages will have new/read markings but PR_CONTENT_UNREAD will
indicate 0 new messages at all times.
.br
When set to 2, PR_CONTENT_UNREAD indicates the number of new messages for the
particular user. (Outlook does not show this number; in Folder Properties, the
radiobox is even greyed out.)
.br
Default: \fI2\fP
.TP
\fBexmdb_private_folder_softdelete\fP
Enables soft-delete support for folders in private stores. (This feature is
experimental.) Public folders always have this on.
.br
Default: \fIyes\fP
.TP
\fBexmdb_schema_upgrades\fP
This directive controls whether database schemas are automatically upgraded
when a mailbox is loaded. During this time, the mailbox is unavailable and
operations on it will be delayed. Connection aborts, if any, would be due to
timeouts in clients rather than servers. (The procedure takes roughly
36sec per gigabyte of exchange.sqlite3 worth of data, or 36sec per about 110k
messages, on a 3700X CPU, single-thread. The file can also temporarily grow to
double its size, so ample disk space may be required.)
.br
Default: \fIyes\fP
.TP
\fBexmdb_eph_prefix\fP
A path for where variadic data files that are process-temporary can be stored.
This may be used to keep the tables.sqlite3 file off an NFS-backed homedir.
Required disk space scales linearly with open table handles and linearly with
messages in the opened folders, at about 80 bytes per messages. (In other
words, don't lump molasses of messages into a single folder in a shared mailbox
read by multiple people.) When you create the directory, set its ownership to
gromox:gromox and mode to 0770.
.br
Default: (empty)
.TP
\fBexmdb_search_pacing\fP
When initially populating a search folder (static or dynamic), yield the lock
on the sqlite database (file descriptor) after so many messages to give other
clients a chance to perform an action.
.br
Default: \fI250\fP
.TP
\fBexmdb_search_pacing_time\fP
When initially populating a search folder (static or dynamic), yield the lock
on the sqlite database (file descriptor) after this much time has passed to
give other clients a chance to perform an action.
.br
Default: \fI2s\fP
.TP
\fBexmdb_search_nice\fP
Run the search folder population thread with adjusted niceness, which affects
process scheduling. This is not an absolute priority as the nice(1) command
would use, but a relative one, as per the nice(2) syscall. The allowed range in
Gromox is 0 .. 19; negative values are not supported (and not meaningful,
because Gromox will usually be running in an unprivileged setting where it is
not possible to raise the priority).
.br
Default: \fI0\fP
.TP
\fBexmdb_search_yield\fP
Make the search folder population thread not only give up the lock on the
sqlite database temporarily, but also invoke pthread_yield(3) after every work
block (cf. exmdb_search_pacing).
.br
Default: \fIno\fP
.TP
\fBexrpc_debug\fP
Log every incoming exmdb network RPC and the return code of the operation in a
minimal fashion to stderr. Level 1 emits RPCs with a failure return code, level
2 emits all RPCs. Note that direct function calls from within the process image
are not logged this way, so this will not show exmdb_provider invocations from
exchange_emsmdb(4gx). Note the daemon log level needs to be "debug" (6), too.
.br
Default: \fI0\fP
.TP
\fBlisten_ip\fP
The IPv6 socket address for exposing the exmdb service on. The address must
conform to gromox(7) \sc "Host addresses".
.br
Default: \fI::1\fP
.TP
\fBmax_ext_rule_number\fP
Default: \fI20\fP
.TP
\fBmax_router_connections\fP
As a exmdb server, permit at most this many inbound connections for the purpose
of sending notifications on these channels. Note that every incoming TCP
connection starts as a data connection and only becomes re-classified as
"notification" once the LISTEN_NOTIFICATION RPC has been issued by the client.
.br
Default: unlimited (only limited by ulimits)
.TP
\fBmax_rpc_stub_threads\fP
As a exmdb server, permit at most this many inbound connections
for commands.
.br
Default: unlimited (only limited by ulimits)
.TP
\fBmax_rule_number\fP
Default: \fI1000\fP
.TP
\fBmax_store_message_count\fP
The maximum number of messages any one particular message store is allowed to
keep. The technical limit is somewhere around 2^47.
.br
Default: \fI0\fP (no limit)
.TP
\fBnotify_stub_threads_num\fP
For every remote exmdb server in exmdb_list.txt, establish and keep this many
number of outbound connections for receiving notification RPCs.
.br
Default: \fI4\fP
.TP
\fBpopulating_threads_num\fP
The number of threads to spawn that will work on asynchronous search folder
population.
.br
Default: \fI4\fP
.TP
\fBrpc_proxy_connection_num\fP
For every remote exmdb server in exmdb_list.txt, establish and keep this
many number of outbound connections for sending commands.
.br
Default: \fI10\fP
.TP
\fBsqlite_debug\fP
If set to 1, every query given to SQLite prepare/execute is logged.
If set to 0, only failed queries are logged. (It cannot be made completely
silent, since our queries ought to never fail.)
.br
Default: \fI0\fP
.TP
\fBtable_size\fP
Maximum number of concurrently active mailboxes.
.br
Default: \fI5000\fP
.TP
\fBx500_org_name\fP
.br
Default: (unspecified)
.SH Multiserver selection map
The SQL column \fBusers.homedir\fP specifies a home directory location in an
abstract namespace. This abstract namespace is shared between all Gromox
programs, and can be used to divide users into custom subsets and steer
connections to different servers.
.PP
exmdb_list.txt specifies how to map from this namespace to exmdb servers. The
file is used by exmdb clients to select the right server to connect to, and the
file is used by exmdb_provider to set up its own data structures.
.PP
Each line in this file consists of 4 columns separated by whitespace:
.IP \(bu 4
Initial prefix to match a user's exmdb home directory on. The pattern should
almost always end in a '/' character, otherwise a prefix of "/home" is able to
match a userdir of "/home2/username" as well, which may be undesired.
.IP \(bu 4
The type of mail stores that are served beneath the prefix. This must either be
"private" or "public".
.IP \(bu 4
The IPv6 socket address of the exmdb server to connect to for all requests
involving this prefix. The address must conform to gromox(7) \sc "Host
addresses".
.IP \(bu 4
The port number.
.PP
In the absence of exmdb_list.txt, two implicit default entries are used:
.RS
.PP
.nf
/var/lib/gromox/user/ private ::1 5000
/var/lib/gromox/domain/ public ::1 5000
.fi
.RE
.SH Network protocol
The transmissions on the socket are simple concatenations of protocol data
units built using the NDR format. The PDU length is present within the PDU
itself near the start.
.PP
.in +4n
.EX
{
	leuint32_t length;
	char pdu[];
}
.EE
.in
.PP
.in +4n
.EX
pdu := {
	uint8_t call_id;
	string directory;
	switch (call_id) {
		...
	}
}
.SH Files
.IP \(bu 4
\fIconfig_file_path\fP/exmdb_list.txt: exmdb multiserver selection map.
.IP \(bu 4
\fIdata_file_path\fP/mail_bounce/
.PP
\fIconfig_file_path\fP and \fIdata_file_path\fP is determined by the
configuration of the program that loaded the exmdb_provider component.
.SH See also
\fBgromox\fP(7), \fBhttp\fP(8gx)
